Crew Decision-Making using Fused S/EVS 1 Running Head: Crew Decision-Making used Fused S/EVS Commercial Flight Crew Decision-Making during Low-Visibility Approach Operations using Fused Synthetic / Enhanced Vision Systems

NASA is investigating revolutionary crew-vehicle interface technologies that strive to proactively overcome aircraft safety barriers that would otherwise constrain the full realization of the next-generation air transportation system. A fixed-based piloted simulation experiment was conducted to evaluate the complementary use of Synthetic and Enhanced Vision technologies. Specific focus was placed on new techniques for integration and/or fusion of Enhanced and Synthetic Vision and its impact within a two-crew flight deck on the crew’s decision-making process during low-visibility approach and landing operations. Overall, the experimental data showed that significant improvements in situation awareness, without concomitant increases in workload and display clutter, could be provided by the integration and/or fusion of synthetic and enhanced vision technologies for the pilot-flying and the pilot-not-flying. During non-normal operations, the ability of the crew to handle substantial navigational errors and runway incursions were neither improved nor adversely impacted by the display concepts. The addition of Enhanced Vision may not, unto itself, provide an improvement in runway incursion detection without being specifically tailored for this application. Existing enhanced vision system procedures were effectively used in the crew decision-making process during approach and missed approach operations but having to forcibly transition from an excellent FLIR image to natural vision by 100 ft above field level was awkward for the pilot-flying. Crew Decision-Making using Fused S/EVS 4 Commercial Flight Crew Decision-Making during Low-Visibility Approach Operations using Fused Synthetic / Enhanced Vision Systems The United States air transportation system is undergoing a transformation to accommodate a projected 3-fold increase in air operations by 2025 (Joint Planning and Development Office, 2004). Technological and systemic changes are being developed to significantly increase the capacity, safety, efficiency, and security for this Next Generation Air Transportation System (NextGen). One of the key capabilities envisioned to achieve these goals is the concept of Equivalent Visual Operations (EVO), whereby Visual Flight Rules (VFR) operational tempos and procedures (e.g., separation assurance) are maintained independent of the actual weather conditions. One approach to attain the goal of EVO would be the creation of a virtual visual flight environment for the flight crew, independent of the actual outside weather and visibility conditions, through the application of Enhanced Vision (EV) and Synthetic Vision (SV) technologies. NASA is investigating revolutionary crew/vehicle interface (CVI) technologies that have the potential to optimize situation awareness and reduce the propensity for, and minimize the risks associated with, pilot error while proactively overcoming aircraft safety barriers that would otherwise constrain the full realization of the NextGen (National Aeronautics and Space Administration, n.d.). Part of this research effort involves the use of EV and SV systems and other interface modalities as enabling technologies to meet the safety challenges of the NextGen EVO operational concept – that is, having the safety and capacity rates of present-day VFR operations in Instrument Meteorological Conditions (IMC). Crew Decision-Making using Fused S/EVS 5 Synthetic and Enhanced Vision SV is a computer-generated image of the external scene topography, generated from aircraft attitude, high-precision navigation, and data of the terrain, obstacles, cultural features, and other required flight information. SV provides significant improvements in terrain awareness and reductions in the potential for Controlled-Flight-Into-Terrain incidents/accidents compared to current cockpit technologies (Kramer, Arthur, Bailey, & Prinzel, 2005; Arthur, Prinzel, Kramer, Bailey, & Parrish, 2003; Schiefele et al., 2005; Schnell, Theunissen, & Rademaker, 2005). EV (or Enhanced Flight Vision System, EFVS) is an electronic means to provide an image of the forward external scene topography by use of an imaging sensor, such as a ForwardLooking InfraRed (FLIR) or millimeter wave radar (MMWR). EV during low-visibility approach conditions provides significant improvements in flight performance, pilot workload reduction, and decreases in the propensity for missed approaches (e.g., see Connor & Mages, 1993). The intended use of EV and SV technology mirror each other as they both attempt to eliminate low-visibility conditions as a causal factor to civil aircraft accidents and replicate the operational benefits of clear-day flight operations, regardless of the actual outside visibility condition. The methodology by which this capability is achieved through SV or EV, however, is significantly different. While some may consider the technologies to be competing; they are, in fact, complementary (Arthur, Kramer, & Bailey, 2005). SV, by virtue of being weather-independent and unlimited in field-of-regard, is particularly advantageous during flight phases, such as approach, which may be obscured by clouds and precipitation of which an EV sensor cannot penetrate. Recognition of terrain and Crew Decision-Making using Fused S/EVS 6 cultural features may also be improved over an EV view since the display presentation is optimized by the display designer, not the product of the sensor and its environment. Pilot recognition of EV terrain and cultural features depends upon the reflected, emitted, and / or refracted energy at the spectral frequencies of the EV sensor and the ability of the pilot to (correctly) interpret this image. Atmospheric effects, time of day, and sensor characteristics can be important factors in the quality of the EV imagery. On the other hand, EV provides a direct view of the vehicle external environment; independent of the derived aircraft navigation solution or of a database. Under conditions of smoke, haze, and night, a FLIR/EV provides orders-ofmagnitude improvement over the pilot’s natural vision; greatly enhancing the pilot’s situation awareness and reducing the pilot’s workload. The comparison of SV and EV, as shown in Figure 1, on a night visual meteorological conditions (VMC) approach into an airfield highlights the similarities and differences in these two technologies. Figure 1. Synthetic vision and enhanced vision comparison. Past Research Previous synthetic vision research (Parrish, Busquets, Williams, & Nold, 2003) has shown that a “flight-critical” synthetic vision implementation which uses automated decision aiding functions for object detection and database alignment/navigation error detection produces superior performance to synthetic vision concepts with an EV inset display. To date, however, Crew Decision-Making using Fused S/EVS 7 technology for “perfect” object detection and database/navigation error detection does not exist. Further, even if these systems come to fruition, there may still be gaps, such as minimal radar cross section objects or below-threshold detection values, which may require other additional integrity and error checks. SV with EV inset displays may offer one possible method to provide the pilot with information sufficient to perform navigation integrity and obstacle clearance checks. While these concepts are viable, performance and pilot workload (Parrish et al., 2003) suffer in comparison to automated methods to achieve these same capabilities. Other studies have shown similar results (McKay, Guirguis, Zhang, & Newman, 2002). Object detection by pilots was found to be best using a dedicated EV display, as opposed to a “shared” display, particularly one that did not include symbology. (However, the presence or absence of symbology was not tested.) From this study and others, the ability of pilots to perform navigation integrity checks and obstacle identification principally depends upon the visual acuity provided by the displayed imagery for the pilot (Boff & Lincoln, 1988), as affected by the resolution and acuity of the sensor and display systems; the characteristics of the object and its surrounding scene or background features; the display clutter; display size; and the display and object color and contrast characteristics. While EV might improve SV operations, the converse warrants investigation as well. In 2004, Section 91.175 of the Federal Aviation Regulations was amended such that operators conducting straight-in instrument approach procedures (other than Category II or Category III) may now operate below the published Decision Height (DH) and Minimum Descent Altitude (MDA) when using an approved EFVS on the pilot’s Head-Up Display (HUD). This rule change now provides “operational credit” for EV. As such, EV operations will become more prevalent. Crew Decision-Making using Fused S/EVS 8 No such credit currently exists for SV. However, SV may be advantageous during flight phases, such as approach, in aiding the pilot’s awareness of terrain, obstacles, and flight path which may be obscured by clouds and precipitation of which an EV sensor cannot penetrate. SV may also provide the crew with “visual momentum” to assist the crew’s understanding and correct interpretation of the EV sensor imagery. Current Study A fixed-based simulator experiment was conducted to evaluate the complementary use of SV and EV technologies, specifically focusing on new techniques for integration and/or fusion of synthetic and enhanced vision technologies and its influence on crew coordination during lowvisibility approach and landing. The objective of this experiment was to test the utility, acceptability, and usability of integrated/fused EV and SV technology concepts in a two-crew commercial or business aircraft cockpit; these results are described in Bailey, Kramer, and Prinzel (2006). The current paper describes experimental results specific to crew decisionmaking during low-visibility approach and crew response to non-normal events that were staged in this experiment using a fused synthetic/enhanced vision system. Method Subjects Twenty-four pilots, representing seven airlines and a major cargo carrier, participated in the experiment. All subjects had previous experience flying with HUDs. The subjects had an average of 1787 hours of HUD flying experience and an average of 13.8 years and 16.2 years of commercial and military flying experience, respectively. EV experience was not required although some pilots were familiar with imaging sensor technology from prior military flight experience. None of the subjects were flying EV in current operations. Crew Decision-Making using Fused S/EVS 9 Simulator The experiment was conducted in the Integration Flight Deck (IFD) simulation facility at NASA Langley Research Center (LaRC). The IFD emulates a Boeing B-757-200 aircraft and provides a full-mission simulator capability. The collimated out-the-window (OTW) scene is produced by an Evans and Sutherland ESIG 4530 graphics system providing approximately 200 degrees horizontal by 40 degrees vertical field-of-view (FOV) at 26 pixels per degree resolution. The OTW imagery used the same source data as the SV database. The subjects occupied the left (as pilot-flying, PF) and right (as pilot-not-flying, PNF) seats. The left seat included an overhead HUD projection unit and the right seat included an auxiliary display (AD) under the right side window. Traditional primary flight and navigation displays were presented head-down.